Method and apparatus for evoking perceptions of affordances in virtual environments

ABSTRACT

Methods and apparatus are provided for evoking perceptions of affordances in a user/virtual environment interface. The method involves recognizing the absence or inadequacy of certain sensory stimuli in the user/virtual environment interface, and then creating sensory stimuli in the virtual environment to substitute for the recognized absent or inadequate sensory stimuli. The substitute sensory stimuli are typically communicated to the user (e.g., visually and/or audibly) as properties and behavior of objects in the virtual environment. Appropriately designed substitute sensory stimuli can evoke perceptions of affordances for the recognized absent or inadequate sensory stimuli in the user/virtual environment interface.

This application is a divisional of application Ser. No. 10/957,288,filed Sep. 30, 2004, status allowed.

TECHNICAL FIELD

The present invention generally relates to virtual environments, andmore particularly relates to a method for evoking perceptions ofaffordances in virtual environments.

BACKGROUND

Virtual (or unreal) environments that are created by interactivecomputer simulation technology can be used in a wide range ofapplications, such as training, education, entertainment, and many othertypes of computer-assisted user-to-environment interfaces. An“immersive” virtual environment is typically designed to provide a userwith the sense of being totally immersed in an artificial,three-dimensional world that is generated by computer software. Forexample, a virtual command and control station environment can becomputer simulated for the purpose of training operators and stationmanagers without the need for a physical mock-up. Virtual environmentsare generally implemented through the use of head mounted displays(HMD), computer screens, or some other type of display device that canbe closely interfaced with the visual receptors of a user. The usertypically interacts with a virtual environment through the use of inputdevices such as mice, joysticks, data gloves, wands, and the like. Theillusion of being immersed in a virtual environment can also be enhancedthrough the use of auditory and other sensory technologies.

For a virtual environment (VE) to be an effective simulation of a realworld environment, the VE should typically provide an immersiveinteractive experience for a user in as realistic an environment aspossible. Recent studies of VE technology, however, have indicated thatthe typical user interface to a VE may be less than optimal. Forexample, in a training application, the VE may not be sufficientlyusable, or may require excess resources to train users, or may notimprove user performance as expected. These shortcomings can lead tocostly and ineffective VE systems.

An improved user/VE interface may enhance the perceptive and interactiveexperience of the user, and could thereby increase the utility andeffectiveness of the VE system. One theory of direct perception, knownas “affordance” theory, can be relevant to VE system design. Affordancetheory is based on the study of the interactions of an organism withobjects in its environment. That is, an affordance can be defined as anintrinsic property of an object or event in an environment as perceivedby a human, and how the human understands what can be done in regard tothe object or event. Since affordances purport to predict the form ofcommunication between objects and observers of an environment, VEdesigns that enable the realization of affordances can improve theuser/VE interface to more closely simulate a real world experience.

Accordingly, it is desirable to provide a method of enabling therealization of affordances in a virtual environment. In addition, it isdesirable to provide design parameter considerations for the realizationof virtual environment affordances. Furthermore, other desirablefeatures and characteristics of the present invention will becomeapparent from the subsequent detailed description and the appendedclaims, taken in conjunction with the accompanying drawings and theforegoing technical field and background.

BRIEF SUMMARY

According to various exemplary embodiments, devices and methods areprovided for evoking perceptions of affordances in a user/virtualenvironment interface. One method comprises the steps of recognizing theabsence or inadequacy of certain sensory stimuli in the user/virtualenvironment interface and then creating sensory stimuli in the virtualenvironment to substitute for the recognized absent or inadequatesensory stimuli. The substitute sensory stimuli are typically generatedby properties and behavior of objects in the virtual environment and aredesigned to exceed a minimum response threshold. The substitute sensorystimuli can therefore evoke perceptions of affordances for therecognized absent or inadequate sensory stimuli in the user/virtualenvironment interface.

An apparatus for evoking perceptions of affordances in a virtualenvironment comprises a processor configured with virtual realitysoftware for generating a virtual environment on a display arrangement.The display arrangement is typically configured as a head-mounteddisplay, a computer screen, or some other display arrangement thatclosely interfaces with the user. An interactive control device such asa mouse or joystick is typically coupled to the display arrangement toenable the user to interact with the virtual environment via theinteractive control device. The processor is further configured togenerate substitute sensory stimuli in the virtual environment to takethe place of recognized absent or inadequate sensory stimuli. Thesubstitute sensory stimuli are designed to evoke perceptions ofaffordances for the recognized absent or inadequate sensory stimuli inthe virtual environment.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will hereinafter be described in conjunction withthe following drawing figures, wherein like numerals denote likeelements, and

FIG. 1 is an illustration of an exemplary model for human perceptionbased on sensory stimuli;

FIG. 2 is an illustration of an exemplary model for the realization andexploitation of affordances;

FIG. 3 is an illustration of an exemplary model of potential failures inrealizing affordances in virtual environments.

FIG. 4 is an illustration of an exemplary embodiment of a model forsubstituting sensory stimuli in a virtual environment to enable therealization of affordances.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the invention or the application and uses of theinvention. Furthermore, there is no intention to be bound by anyexpressed or implied theory presented in the preceding technical field,background, brief summary or the following detailed description.

Various embodiments of the present invention pertain to the area ofvirtual environment (VE) interface design. Where it is determined thatcertain sensory stimuli are absent or inadequate in a VE, substitutionalsensory stimuli can be provided to enable the realization of affordances(sensory cues) to the user. As a result, the perceptive and interactivecapabilities of the user can be enhanced through the use of substituteaffordances in the VE. By improving the user/VE interface in thismanner, the user can perceive a more faithful representation of the realworld environment that is being simulated.

A typical VE interface design aims to present a virtual world, in whichusers can experience a strong sensation that they are present in, orpart of, a computer-generated world. For example, a head-mounted display(HMD) can be used to present an immersive type of VE to a user.Typically, the HMD can be worn as a helmet and is generally equippedwith a graphical display in the form of tiny computer screens configuredas goggles. The HMD can also be equipped with other sensory devices suchas earphones. Other VE configurations in current use include varioustypes of boom-mounted head coupled displays, computer screenarrangements and projection systems. The VE display device, such asdisplay 408 of FIG. 4, is typically provided with video and audio inputsignals generated by virtual reality software running on an externalprocessor, such as processor 406 of FIG. 4. In addition, the user istypically equipped with some type of input device such as a mouse orjoystick, or interactive control device 410 of FIG. 4, that enables theuser to interact with the VE. Display 408 and Interactive control device410 may be connected to processor 406 and to virtual environment 304 ofFIG. 3. Processor 406 may be connected to virtual environment 304.

The concept of “immersiveness”, while generally engaging and vibrant,brings with it additional design issues that are not present inconventional, non-immersive, human-to-computer interaction (HCI).Existing HCI design principles typically focus on staticrepresentations, and often fail to fully incorporate theories ofperception appropriate for the dynamic multimodal interactions inherentin a user/VE interface. The use of existing HCI design principles canlead to VE designs with less than optimal utility, because the users maynot readily perceive the actions and functions that can and should beenacted, for example in a learning application. Therefore, it isgenerally desirable to integrate a comprehensive theory of perceptioninto VE design.

One perception theory generally believed to be relevant to VE systemdesign is known as affordance theory. Affordance theory is based on theinteraction of an organism with its environment. Affordances may bedefined as the inherent uses that an object in an environment furnishesto its user. As such, affordances may be thought of as a form ofcommunication between objects and their users. Therefore, VE designsbased on affordance theory can help bridge the gap between a typical HCIdesign and a truly immersive VE design by providing selected affordancesto the user/VE interface that more closely align the perceptions of auser to those that are typically experienced in a real worldenvironment.

An exemplary model 100 of perceptions and interactions of a typicalhuman 102 with an environment 104 is illustrated in FIG. 1. In thisgeneral overview, environment 104 may be natural (real world) orsynthetic (virtual). Objects 106 within environment 104 are assumed tohave characteristic properties and behaviors. Such properties andbehaviors represent typical external stimuli for the sensory perceptionof human 102. These external stimuli can be considered as forms ofenergy reflected or transmitted at the object. When measured at theobject, an external stimulus is typically categorized as a distalstimulus, and when the energy reaches a sensory receptor (e.g., theeye), it is typically categorized as a proximal stimulus.

A sensory receptor that is adapted for transducing energy from anexternal stimulus (i.e., outside the body) is typically categorized asan exteroceptor. In the FIG. 1 model 100, a vision exteroceptor 108 andan audition exteroceptor 110 represent typical sensory receptors ofhuman 102 for perceiving light and sound, respectively. Sensoryreceptors that respond to direct contact with external stimuli (e.g.,pressure/temperature, taste, smell) are typically categorized asproprioceptors. In the FIG. 1 model 100, cutaneous proprioceptor 112,gustation proprioceptor 114 and olfaction proprioceptor 116 representtypical sensory receptors of human 102 for perceivingpressure/temperature, taste, and smell, respectively.

In addition to the external stimuli described above, human 102 typicallyexperiences internal stimuli as well. Sense organs that respond tointernal stimuli within the body are typically categorized asinterceptors. Examples of typical interceptors are shown in the FIG. 1model 100 as a kinesthetic interceptor 118, a vestibular interceptor 120and an organic interceptor 122, representing sensory receptors of human102 for internal stimuli such as body motion, balance, andthirst/hunger, respectively.

As the sensory receptors of human 102 are stimulated (stimulusintegration 124), the receptors generally emit mediated stimuli in theform of nerve impulses moving to the central nervous system from theperipheral nervous system (a process called affection). This perceptionprocess is generally completed when human 102 forms a “percept”, meaningthat which is perceived. This percept forms a basis for other cognitiveactivities such as decision-making and execution 126. For example, adecision can be made to select and exercise various environmentaleffectors, such as a hand 128, an arm 130, or a leg 132.

Through such decisions and resulting actions, human 102 can influenceenvironment 104, either by changing the environment itself or theposition of human 102 within environment 104. As noted above, human 102can interact with environment 104 by using a mouse or a joystick orother interactive input device (not shown). The observation of thestimuli-decision-action cycle by human 102 can engender a feedback cyclethat also affects perception. As such, the specific environmentaleffectors available (e.g., hand 128, arm 130, leg 132) and the relatedexperience of human 102 can influence the percepts formed. For example,human 102 can perceive that a surface is suitable for walking because ofthe stimuli the surface presents to the sensory receptors of human 102,and because human 102 has the availability of environmental effectors(legs 132) that are capable of walking.

The knowledge of and experience with environmental effectors typicallyenables human 102 to understand the available action capabilities forusing those effectors, such as walking, crawling, running, and the like.Similarly, additional internal state information such as current goals,motivations, and knowledge of stature (e.g., size, reach, etc. of human102) can also influence perception. Moreover, for a percept to beformed, such internal state information and environmental sensorystimuli must generally reach a certain intensity level and timeduration. That is, some stimuli may have insufficient intensity orduration to activate a response in a particular sensory modality (e.g.,sight, hearing) and therefore may fall below a minimum responsethreshold for the sensory receptor involved. Minimum response thresholdsgenerally vary for each sensory modality, and may also vary fordifferent sensory receptors within a modality, e.g., rod and cone cellsin vision receptors. Stimuli failing to rise above the relevant minimumresponse threshold may not lead to new percepts.

As noted previously, affordance theory describes how humans directlyperceive objects in their environment. For example, a chair typicallyaffords sitting to humans. In addition, a chair can provide otheraffordances, such as standing, climbing, blocking, and so forth. If thegoal of a human is to sit, the human typically looks for certainrelevant properties of the chair. For example, humans generally want aseat that is flat, level, and knee height, and may also seek otherproperties such as comfort padding. There is generally an acceptabilityrange for each property as well. That is, the seat may not need to beperfectly flat, or exactly at knee height, for example. Humans aregenerally known to be able to instantly adapt to a wide variety ofseats, including chairs never sat on and chair designs never beforeseen. Therefore, a chair within an environment typically providesaffordances, or potential uses, that a human can readily perceive.

Since affordances can be used to describe how humans operate and adaptto a real environment, a VE design based on affordance theory can offera user/VE interface that behaves in a more understandable and reliableway than a VE design that is not sensitive to its affordances. Researchstudies of affordance theory generally conclude that affordances arerealized through the integration of environmental stimuli and internalstate stimuli interacting with experience gained throughdecision-making, which in turn affects the human knowledge of internalstate, action capabilities, and body stature in the environment. Thisconcept is illustrated in the exemplary model 200 of FIG. 2, whichdepicts the realization and exploitation of affordances in a typicaluser/VE interface.

The stimulus integration block 124 of FIG. 1 is superseded by the directperception of affordances block 201 of FIG. 2. In addition, a feedbackloop of blocks 202, 204, 206, 208 is connected betweendecision-making/execution block 126 (FIG. 2) and direct perception ofaffordances block 201. The feedback loop represents the interaction ofexperience 202 gained through decision-making with state stimuli such asinternal state 204, action capabilities 206, and body stature 208. Inthe exemplary model 200 of FIG. 2, internal state 204 examples typicallyinclude goals, motivation and spatial knowledge. Similarly, actioncapabilities 206 examples typically include sitting, walking, running,grasping and swinging, while body stature 208 examples typically includerelative physical dimensions and an egocentric perspective.

Direct perception of affordances block 201 will typically integrateexternal stimuli from objects 106 (via extroceptors 108, 110, andproprioceptors 112, 114, 116), with internal stimuli (via interceptors118, 120, 122) and with state stimuli (via internal state 204, actioncapabilities 206, and body stature 208) to enable the completion of apercept for cognitive activities such as decision-making and execution.As in the FIG. 1 model 100, a decision can be made (block 126) to selectand exercise various environmental effectors (128, 130, 132).

Affordances may not be correctly realized in all circumstances, however,even in a natural environment. For a designed artificial environmentsuch as a VE, certain affordances may be absent or impoverished and maynot support the realization desired by their designers. In general, abasic source of failure to realize an affordance is typically aninsufficient sensory stimulus; i.e., one below the minimum responsethreshold. For example, a failure of the proximal stimulus to activatethe sensory receptors could result from an insufficient intensity leveland time duration of the proximal stimulus. Other potential sources ofaffordance realization failure can include the absence of a desiredlevel of multimodal sensory stimulation and/or an inadequate perceptionof action capabilities or body stature.

The model 300 illustrated in FIG. 3 is intended to represent potentialfailures in realizing affordances at a user/VE interface. For example,the following types of affordance realization failures are among thosethat may occur between human 102 and a VE 304:

-   -   1) Insufficient sensory stimuli;    -   2) Insufficient multimodal sensory stimulation;    -   3) Inadequate perception of body stature in the environment; and    -   4) Inadequate perception of action capabilities in the        environment.        In FIG. 3, exemplary model 300 is shown with insufficient        stimuli sources, as represented by shaded proprioceptors 112,        114, 116, shaded interceptors 118, 120, 122, shaded action        capabilities 206, and shaded body stature 208. As such, the        arrows representing these insufficient stimuli have been        eliminated between the shaded sources and direct perception of        affordances block 201. As a result of these missing stimuli        inputs to direct perception of affordances block 201, the        percept generated by block 201 will generally be deficient for        optimal decision making in block 126. For example, if human 102        is expected to pass through a virtual doorway in VE 304, the        absence of body stature 208 and action capabilities 206 stimuli        could prevent human 102 from knowing how to (virtually) navigate        through the doorway due to the unrealized affordances.        Similarly, the absence of proprioceptor stimuli (112, 114, 116)        and interceptor stimuli (118, 120, 122) will typically impair        the optimal interaction of human 102 with VE 304.

Since a VE represents a kind of reality that seeks to immerse a userwithin that reality, it is desirable that the VE behaves like reality orin easily understandable deviations from reality. Therefore, thepotential breakdowns in the realization of affordances in a VE should beminimized to the greatest extent possible. To this end, studies havebeen made to determine methods of overcoming these potential breakdowns.Three possible methods are:

-   -   1) correlating sensory stimuli to the experience represented in        the virtual world;    -   2) determining that a particular modality is irrelevant to the        experience; or    -   3) substituting sensory information via another modality for        missing modalities.        Of these, the latter approach, sensory information substitution,        appears to have the greatest potential for success since it is        generally not limited by current technology (as the first        approach may be), and generally does not involve diminishment of        an experience (as is likely with the second approach). As such,        sensory substitution methods, to be described below, can be used        to replace missing stimuli in a VE in order to evoke the        realization of affordances.

According to an exemplary embodiment of a sensory substitution method400, as illustrated in FIG. 4, substitution stimuli 402, 404 can becommunicated through the properties and behaviors of objects 106 thatare represented in VE 304. In general, successful sensory substitutionstimuli should be designed to:

-   -   1) replace sensory modalities not represented in the virtual        environment;    -   2) outweigh natural stimuli not correlated with the virtual        environment; and    -   3) exceed minimum response thresholds in the modality used for        the substitution.

Since the substitution stimuli are typically constrained to theproperties and behaviors of objects in the virtual environment, thesensory receptors likely to be exploited are the exteroceptors and/orthe proprioceptors. Generally, the more effective and convenientreceptors for substitution stimuli schemes are the exteroceptors,representing the visual and auditory senses. Therefore, designparameters for substitution stimuli typically include:

-   -   1) the sensory modality (e.g., visual or auditory);    -   2) the magnitude of the sensory stimuli (e.g., size of a visual        object, loudness of an auditory stimulus); and    -   3) the optimal form of the sensory stimuli (e.g., a projected        human shape).

Referring again to FIG. 4, substitution stimulus 402 representssubstitutional cues of action capabilities to replace the missingaffordances from shaded action capabilities block 206. Similarly,substitution stimulus 404 represents substitutional cues of body statureto replace the missing affordances from shaded body stature block 208.The aforementioned substitutional cues (402, 404) can be communicatedvia objects 106 to exteroceptors 108 of human 102. In the aforementionedvirtual doorway example described in conjunction with model 300 in FIG.3, human 102 may have difficulty negotiating the doorway because of themissing affordances from action capabilities block 206 and body statureblock 208. However, by adding substitutional cues 402, 404 to VE 304, asillustrated in FIG. 4, action capabilities and body stature affordancescan be communicated via objects 106 and exteroceptors 108, 110 to directperception of affordances block 201 of human 102. As a result, human 102can better judge the relative size of the virtual doorway with respectto body size, and the action a body (as represented by a virtual figure)should take to navigate through the doorway.

Substitutional cues can take various forms depending on the affordanceto be evoked. For example, one type of spatial affordance can bedesignated as a passability affordance; e.g., a doorway or passageway,or the like. An exemplary passability affordance can be evoked by theappearance of objects in a VE with appropriate gaps and edges. Moreover,the physical characteristics of the object can be communicated by visualcues relating the size of the object (e.g., shoulder width of thedoorway/passageway) to the virtual representation of the human. Inaddition, an action capability can be provided as a (virtual) capabilityto rotate. A practical example of a passability affordance in a VEaircraft maintenance trainer might be the accurate realization of a userthat he can experience the same limitations on access in the VE as hewould in the real aircraft.

In another example, a temporal type of affordance can be designated as acatchability affordance. Substitutional cues for evoking an exemplarycatchability affordance may take the form of visual and audio stimuli inthe VE that indicate detectable height and speed. Moreover, thesubstitute stimuli may also provide visual and audio cues that indicateself-acceleration and self-speed when objects to catch will land withina predetermined range. In addition, an action capability can be providedas a (virtual) capability to move. In a practical example, a VEinterface implementing training tools for air traffic controllers wouldtypically require an accurate realization of catchability by theoperator to ensure that the closure rates of different aircraft arecorrectly understood.

Accordingly, the shortcomings of the prior art have been overcome byproviding an improved method for evoking perceptions of affordances in avirtual environment (VE). A typical VE may not provide all of thesensory stimuli to evoke user perceptions of affordances as would beexperienced in the real world environment simulated by the VE. In orderto improve the VE simulation, therefore, substitute stimuli may becommunicated to the user via the properties and behavior of objects inthe VE. If properly designed, the substitute stimuli can evoke thedesired perceptions of affordances to enhance the user interaction withthe VE.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or exemplary embodiments are only examples, and arenot intended to limit the scope, applicability, or configuration of theinvention in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the exemplary embodiment or exemplary embodiments. Itshould be understood that various changes can be made in the functionand arrangement of elements without departing from the scope of theinvention as set forth in the appended claims and the legal equivalentsthereof.

What is claimed is:
 1. An apparatus for evoking perceptions of affordances in a virtual environment, the apparatus comprising: a processor configured with virtual reality software for generating the virtual environment; the virtual environment comprising a display arrangement in communication with the processor, the display arrangement configured to interface with a user, wherein the display arrangement comprises at least one of a head-mounted display, computer display screen, boom-mounted head-coupled display device, and projection system; and an interactive control device in communication with the virtual environment, the interactive control device configured to enable the user to interact with the virtual environment, wherein the processor is further configured to generate substitute sensory stimuli for recognized absent or inadequate sensory stimuli in the virtual environment, wherein the substitute sensory stimuli comprise at least one of visual cues and auditory cues, wherein the visual cues evoke spatial affordances, wherein the visual cues and the auditory cues evoke temporal affordances, wherein the substitute sensory stimuli exceed a minimum response threshold for user perception, and wherein the recognized absent or inadequate sensory stimuli relate to at least one of properties and behavior of objects in the virtual environment, body stature of the user, and action capabilities of the user. 